Bing vs Google: 11 Issues with the Bing It On Search Survey

In a move reminiscent of the eighties Pepsi Taste Tests, Bing (UK) has released a survey claiming users have picked their search engine above Google’s in a blind side-by-side comparison of equivalent SERPs. I got this email in my inbox recently:

We’re told that, not only are Bing’s search results more popular than Google’s, but also that we can test it ourselves at www.bingiton.com

11 Reasons the BingItOn Survey makes no sense

Now, like most people in the search industry who don’t work for a search engine, I think strong competition in the market is not just healthy, but essential. Also, I think most people who know the search industry will be thinking that something seems not quite right about that claim, so let’s take a look into this claim and see what we can turn up. I am quite prepared to be ‘pleasantly surprised.’

1. Bing’s comparison leaves out the areas of the SERP where Google outperforms them most.

The first thing that sets alarm bells ringing for me is this small print at the bottom of the email:

So the comparison search results they used were:

“Based on an unbranded comparison of the web search results pane.”

Well, that sounds fair enough, a big brand logo would clearly bias people’s opinion. However, the test also:

“excludes all ads.”

This may be fair, and if we’re comparing just organic search, it would clearly be odd to include them, but it would be hard to argue that the ads presented on a SERP aren’t part of the overall search experience inasmuch as they might make one set better or worse than another. What’s more the survey also excluded Google’s Knowledge Graph, which does seem a little like asking people to choose which tin of baked beans they prefer after all the sauce has been removed.

Compare these two sets of search results – I practically dare you to tell me you don’t genuinely prefer the Google set.

(For any younger folks reading this, a CD is an obsolete form of storage for musical data that was still popular when Razorlight was.)

Well, it’s understandable, though, that we might want to compare similarly structured results, and indeed Bing does say:

“To keep it an apples-to-apples comparison of just the algorithmic web search results, any ads or other features were also removed.”

We may come back to this later…

2. The survey claims are based on a very limited subset of searches.

The small print also states that the survey results are based on the most popular searches and links to a site ( http://www.bing.com/blogs/site_blogs/b/uk/archive/2013/10/10/biouk.aspx ) where we can find out a bit more.

Bing says (all my Bing quotes are from the above link or the Bing it On site itself)

“Participants were each asked to do 10 searches, drawn from a list of 450 from the Google UK Zeitgeist 2012 list and a Google trends list from June 2013. This way, we could test searches that most people were likely to have heard of (because they are popular).”

Bing is telling us that these searches are popular, and therefore typical of the kinds of searches people do most often.

Now superficially that might sound fair enough, but of course we know that the great volume of search is in the long tail — the obscure things that interest us, the combinations of popular terms with locations and modifying adjectives, and those weird and wonderful searches done each and every day for the first and only time.

So a popular search is not necessarily representative of what people are actually searching for, and again, the long tail is one of the areas where, personally, I think Bing struggles most.

3. The survey claims are based on a very weird subset of searches.

Now these 450 ‘popular’ searches were taken from where, again?

“the Google UK Zeitgeist 2012 list and a Google trends list from June 2013″

So lists which are popular, yes, but not necessarily the most popular. In fact, these two lists give greatest prominence to trending topics and spikes in traffic for previously lower-volume search terms.

In terms of what this means for Bing’s survey, we have some more issues here, because clearly these trending topics and short term spikes have a tendency to be news content, sports content, social media and viral activity and other things like that, where the ‘ten blue links’ type of search is weakest, and searchers are likely to be looking for the knowledge graph or onebox-type result which gives you an instant answer to your informational need.

And the second issue here is that these searches, which were very relevant at some point in the past, don’t have the same meaning or urgency that they did when they spiked. To take the top sports result from Zeitgeist 2012 as an example, someone searching for ‘synchronized swimming’ today is likely looking for a different type of result to someone making that same search in summer 2012, while the Olympics were in full swing.

This raises another question. Are people rating these results based on what they thought would have been reasonable results back when ‘Hurricane Sandy’ or ‘Olympic torch route’ were newsworthy searches, or more fact-filled reference articles that might be a better answer for those queries today? We aren’t told this.

Let’s Bing It On, then

So it’s probably time to actually have a go at Bing’s Challenge.

The Bing It On homepage offers the bold statement that:

But as we have seen

4. The searches compared aren’t the most popular searches.

They were just the most trendingest searches some months back. And yes, that’s totally a word.

Time for a search then. First up, a vanity search, I’ll try my Twitter handle:

5. The results sets don’t actually look the same at all.

Straight away, we can see that there are differences in layout between the two sides. The typefaces used are different and the way they display the blue link for the same site varies (i.e. one of them isn’t necessarily using the